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ABSTRACT 

Available methods for collecting human factors data 
rely heavily on observations, interviews, and questionnaires. A need 
exists for other methods. The feasibility of using two-way 
voice-radio for this purpose was studied. The data collection 
methodology consisted of a human factors analyst talking from a radio 
base station with technicians wearing portable radio units while they 
worked on jobs dispersed over an area. Verbal communication probed 
for information on equipment problems, procedural problems, delays, 
and potential hazards. The concept of using radio equipment to 
collect human factors data was tested, using technicians performaing 
flight-line maintenance. Data collected by radio were compared with 
data collected by questionnaire. It was concluded that voice-radio is 
not only a feasible means of collecting human factors data in the 
field, but has certain advantages over questionnaires in determining 
hardware problems and amplifying the data-collection capability of 
the human factors analyst. (Author) 
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ABSTRACT 



Available methods for collecting human factors data rely heavily on obser- 
vations, interviews, and questionnaires. A need exists for other methods. The 
feasibility of using two-way voice-radio for this purpose was studied. The data- 
collection methodology consisted of a human factors analyst talking from a radio 
base station with technicians wearing portable radio units while they worked on 
jobs dispersed over an area. Verbal commimication probed for information on 
equipment problems, procedural problems, delays, and potential hazards. The 
concept of using radio equipment to collect human factors data was tested, using 
technicians performing fli^t-line maintenance. Data collected by radio were 
compared with data collected by questionnaire. It was concluded that voice- 
radio is not only a feasible means of collecting human factors data in the field, 
but has certain advantages over questionnaires in determining hardware prob- 
lems and amplifying the data-collection capability of the human factors analyst" 
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SECTION I 
INTRODUCTION 

1. THE PROBLEM 

The process of collecting data regarding man's performance is a significant 
portion of the human factors technology. Data describing work activities of man 
in test and operational environments are needed to allow human Actors special- 
ists to decide, for example, whether the design of an equipment is adequate, 
training is thorough, personnel manning is sufficient, and task procedures are 
adequate and efficient. 

However, the metiiodology available to collect such data is limited. Meister 
and Rabideau (Reference 2, 1965, page 224) concluded that . . most of the 
methods available . . . are fairly unsophisticated, relsring to a considerable ex- 
tent on observation and interview. " Other investi^tors (Keenan, Parker, 
Lenzycki, Reference 1, 1965, and Newton and Askren, Reference 3, 1968) found 
essentially the same thing, reporting that the three methods — observation, 
interview, and questionnaire ~ are the most frequently used means of collecting 
data concerning man's performance In system environments. Based upon these 
analyses of the state-of-the-art in data collection on human performance, and 
based upon the substantial need for data of this type, it was concluded that 
research oriented toward developing new data-collection methods was in order. 

2. PURPOSE OF THE RESEARCH 

In view of the state-of-the-art in data collection, the general purpose of the 
research was to study the feasibility of using two-way voice-radio as a means of 
collecting human performance data. A methodology titled Personnel Activity 
Analysis Radio System (PAARS) was conceptualized. The methodology utilized 
a radio base station operated by a human factors analyst, a number of small 
two-way radio units worn by technicans while working on the job, and voice 
communication between personnel to identify, interpret, and transmit desired 
information and data (Figure 1). 
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The specific objectives of the research were to (a) determine procedures 
suitable for collecting data by voice radio, (b) determine the value of the data 
collected by voice radio, and (c) determine adaptability to the operational envi- 
ronment of this method of collecting human-performance data. Value was defined 
as the kind, quality, and quantity of data collected. Adaptability was defined as 
the worker acceptance and attitude toward the data-collection process, the quan- 
tity of working personnel that could be monitored by the system, and the range 
and shielding limitations of the radio hardware. Finally, since the results of 
this study would be partially a function of the radio equipment that was selected 
for use, an additional objective was established: (d) develop specifications for 
an optimum radio data collection system. 

3. HARDWARE CONFIGURATION 

The conc^t illustrated in Figure 1 was translated into the hardware con- 
figuration illustrated in Figures 2 and 3. Figure 2 shows the base station which 
included a 5-watt transmitter/receiver, a selective call unit which allowed 
conversation with individual technicians, and a tape recorder which recorded 
incoming and outgoing messages. Figure 3 shows the portable radio imit. All 
equipment were stock items and are identified and described in Appendix B. 
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SECTION n 
METHOD 

1. GENERAL 

The diversity of objectives required a mixed approach to the research. 
Therefore, an exploratory test phase and a controlled-experimental test phase 
were conducted. The exploratory test was a limited operation which collected a 
small quantity of data and probed the limits of PAARS data collection capability. 
This test yielded data related to the research objectives of determining voice- 
radio procedures, adaptability of voice- radio data collection in an operational 
environment, acd specifications for an optimum system. The controlled- 
experimental test was a more precise study related primarily to the research 
objective of the value of data collected by voice radio. It also provided some 
information regarding the research objectives of adaptability and optimum 
specifications. 

2. EXPLORATORY TEST 

The exploratory test studied several different data-collection procedures, 
investigated the maximum number of workers wearing portable radios that one 
human factors analyst could monitor from the base station, and studied radio 
equipment range and shielding problems. Radio range and shielding were checked 
by placing one member of the research team equipped with a portable unit in a 
variety of locations at a variety of distances from the base station and deter- 
mining subjectively the quality of voice transmission between the unit and base 
station. 

The maximum number of workers that could be monitored was studied by 
equipping various quantities of Air Force flight line maintenance technicians 
with portable units, and determining subjectively the overload point for voice 
communications aboixt job activities between them and the base station human 
factors monitor. Also, two data-collection procedures were tested and subjec- 
tively evaluated. The procedures were (1), the base station called the mainte- 
nance technician on a time-sampling plan and (2), the maintenance personnel called 
the base station at planned work intervals. 
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3. CONTROLLED-EXPERIMENTAL TEST 

The controUed-experimental test was designed primarily to determine the 
value of the PAARS method as compared with a standard human factors data- 
collection technique, in this case, paper-and-pencil questionnaire. The statis- 
tical model used was a three-way fixed factor analysis of variance with repeated 
measures on all factors (Winer, Reference 4, 1962), 

a. Independent Variables and Procedures 

The experimental design was implemented with data collected from six 
subjects under two conditions (radio and questionnaire), with each condition 
repeated three times. The data consisted of subject responses to the thirteen 
questions listed in Table I. The questions were prepared to cover the categories 
of equipment and tool design, s\fety, logistics and support, training, and work 
procedures, 

TABLE I 

QUESTIONS ASKED MAINTENANCE TECHNICIAN SUBJECTS 

1. What changes in equipment design would make it easier to maintain? 

2. What changes would make it easier to find the test points? 

3. What are the possible problems with safety on this job? 

4. What did you do to get into the equipment to be repaired? 

5. Would a new man have problems with color coding or labelling on 
this equipment? 

6. What were some of the delays on this job? What caused the delay (s) ? 

7. What were the problems with the tools or test equipment on this job? 

8. Could you use different or more tools or test equipment on this job ? 

9. Is there any special training that would help make this job easier? 

10. If you had to train a new man, what would be the most important 
thing for this job ? 
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TABLE I (Continued) 



11. What changes could be made in the T.O. ior this job? 

12. Did you see anything in the 'i.O, chv^cklist that is (was) confusing? 

13. What kind of mistake in procedure is most liktly to happen to a new 
man on this job? 



Experienced Air Force maintenance personnel served as subjects while they 
carried out their regular work assignments. These subjects normally performed 
several electronics maintenance jobs each day. One of these assignments, usually 
the first job of the day, was used as the work sample to be measured by either 
the radio or questionnaire condition. 

During three of the test days, each subject wore a radio unit and answered 
the questions via radio link while at the job site. On three other test days, each 
subject did not wear a radio unit but reported into the PAARS base station in 
person at the end of the day and completed a questionnaire. This end-of-day 
questionnaire (EODQ) was made up of the questions listed in Table I, changed 
only to reflect past tense instead of present tense for some of the questions. The 
experimental conditions, PAARS and EODQ, were used alternately for each sub- 
ject in order to minimize sequential effects. 

b. Dependent Variable and Quantificatior 

The major dependent variable was defined as richness or utility of question 
responses for application to human factors test and evaluation decisions. Quan- 
tification of utility was provided by judgments produced by seven judges expe- 
rienced in human factors analysis, design, test, and evaluation. Each judge was 
instructed (Appendix A) to view at one time all responses (n = 36) to each 
separate question. Each response was typed on a separate sheet of paper. This 
set of 36 responses consisted of the responses in random order of all subjects 
to one question for three maintenance tasks in which the PAARS method was 
used, and for three tasks for which the EODQ method was used. 
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The judges then chose one response as most useful and one response as 
least useful. The respective value of most or least was then indicated by marking 
the appropriate extreme end of a 100-millimeter linear scale provided at the 
bottom of the response sheet. Given these extreme values, each judge then 
assigned to the remaining 34 responses » a relative value (mark on the line) 
ranging between the extreme values. This procedure was followed for each of 
the 13 questions. 

At the completion of judgments, the scale markings were translated to 
numbers ranging from 1 (poor) to 100 (good). From each of the seven judges, 
there were obtained 468 scale values reflecting six subjects, six maintenance 
jobs, and 13 questions. 

A preliminary reduction in data was performed before the analysis was 
carried out. Since there occurred for each subject, three instances of the PAARS 
method and three instances of the EODQ method, the highest value was selected 
from the three jobs under each method for each question. This yielded 26 values 
for each subject from each judge. It was upon these values that the data analysis 
was performed. 

c. Data Analysis 

Within the three-way fixed factor analysis of variance, the principle factors 
of interest were those of method (PAARS versus EODQ) , and questions (questions 
1 - 13). Study of the data regarding these factors determined the kind and quality 
of data collected by the two methods. The design also permitted analysis of con-- 
sistency of judgments across judges (judges 1-7), and interactions between any 
two or three of the factors. The quantity of data collected was determined by 
count of the frequency of responses judged to be at least of some value (any 
rating greater than 1). 
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SECTION III 
RESULTS AND DISCUSSION 

The results are presented in relation to the research objectives of proce- 
dures suitable for voice-radio data-collection, adaptability of the voice-radio 
data-collection process to an operational environment, value of the data collected, 
and specifications for an optimum voice-radio system. 

1, SUITABLE VOICE-RADIO PROCEDURES 

Of the two procedures studied during the exploratory phase, the maintenance 
technician call-in mode was considered superior. The nature of a maintenance 
Job, with unscheduled activity periods, demonstrated immediately the superiority 
of depending upon the technicians to call-in when meaningful or critical actions 
occurred. Sampling the work day of the technicians provided comparatively little 
information useful to system evaluation decisions. Therefore, (he technician 
initiated mode was adopted for use in the controlled-experimental phase that 
followed. 

2. ADAPTABILITY OF VOICE-RADIO TO AN 
OPERATIONAL ENVIRONMENT 

The exploratory test phase established that this off-the-shelf radio system 
configuration was useful and acceptable. The clear communication range was 
three to four miles over open terrain, and at least one mile through multiple 
shielding of hangar wall and aircraft metal skin. Maintenance technicians did 
not react adversely to wearing and using the portable units while working on the 
job, although occasionally, a technician removed his radio when climbing into 
narrow confines of an aircraft. The human factors analyst operating the base 
station monitored the work of six technicians in depth, and ten technicians in a 
more limited way. This information influenced the design of the controlled- 
experimental phase to include six subjects. 
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3. VALUE OF DATA COLLECTED 

The value of the data was determined by analysis of the technician responses 
to the thirteen questions. These results are described in ternis of the main 
analysis of variance, the kind of data best collected by PAARS versus EODQ, the 
overall amount of data collected by radio versus questionnaire, and the overall 
quality of data collected by PAARS versus EODQ. 

a. Main Analysis 

A summary of the main analysis is shown in Table IL Variables found 
significant at the 0.05 level were judges, questions, and the interaction of judges 
with questions. These effects are illustrated in Figures 4, 5, and 6. The factors 
found not significant included methods (PAARS versus EODQ), methods inter- 
action with questions, and judges interaction with methods. Thus, based on this 
overall analysis of the variance of quality of responses, the voice-radio method 
was not significantly different from the questionnaire method. 

The very large F-ratio obtained for the effect of judges clearly indicates 
that the human factors specialists who judged the responses did not find common 
agreement as to their (responses) relative value. This may indicate a weakness 
in the judgment procedure established ia this research. Perhaps a more struc- 
tured criterion for judges is needed, one that relates to specific jobs within 
specific subsystem operations. Certainly, the large judge-variability reduced 
the chance of finding a significant difference between the PAARF and EODQ 
methods. 

The significant variance of questions suggests the importance that must be 
attached to content, structure, and style of questions if questions are to elicit 
comparable quality of ii\formation. 

The significant interaction of judges and questions suggests that the judges 
did not maintain a consistent mental set as to the value of responses, but rather 
shifted their concept of response utility according to the kind of question from 
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TABLE II 

ANALYSIS OF VARIANCE OF QUESTIONS RESPONSES 



Source 


DfioTfies of 
Freedom 


M6311 

Square 


F Value 




1 


1,620 




(Radio vs Questionnaire) 








Judges (J) 


6 


11,812 


16.97* 


Questions (Q) 


12 


17,206 


2.47** 


Subjects (S) 


5 


19,239 




M X J 


6 


498 




M X Q 


12 


6,023 




M X S 


5 


7,457 




J X Q 


72 


1.108 


2.76* 


J X S 


30 


696 




Q X S 


60 


6,956 




M X J X Q 


72 


384 




M X J X S 


30 


471 




M X Q X S 


60 


3,623 




J X Q X S 


360 


402 




M X J X Q X S 


360 


361 








19,239 




* = Significant at< 0.01 
** = Significant at < 0.05 



which the response was obtained. Again, there is indicated a need for a more 
restricted criterion. For example, questions relating to training might be better 
assessed from a different set of rules than questions relating to work procedurest 
tools, and equipment. 

A check on the appropriateness of the use of high-value responses in the 
analysis was made by analyzing median value responses. The significant effects 
found by this computation were identical to those of the high- value data analysis. 
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b. Kind of Data Collected PAARS Versus EODQ 

The thirteen questions used by both PAARS and EODQ methods covered a 
variety of topics. It was reasonable to expect that the radio method was more 
valuable for collecting data for some topics, and the questionnaire more valuable 
for others. However, the data regarding this^point yielded mixed results. As 
indicated in the main analysis of variance of the quality of the data (Table II), 
there was no statistically significant interaction between questions and methods. 
However, Figure 7 shows that the proportion of time some useful data was ob- 
tained was significant* for the radio method for questions 1, 2, and 7, and for 
the quectionnaire method for question 12. Thus, voice-radio seems to yield a 
greater quantity of data regarding equipment and tools, and the questionnaire a 
greater quantity regarding technical order checklists. This trend is supported 
by the interaction effect between methods and questions of the main analysis of 
variance illustrated in Figure 8. Although statistically not significant, these data 
show the PAARS method collecting better quality data regarding equipment and 
tools (questions 1, 2, 4, 5, and 7) than the questionnaire method. 

c. Overall Amount of Data Collected PAARS Versus EODQ 

All radio responses were judged on the average tc have at least some utility 
49.3% of the time. All questionnaire responses were judged on the average to 
have at least some utility 48.5% of the time. These results indicate that both 
methods were reasonably productive means of collecting information with neither 
practical nor statistical difference between methods. 

d. Overall Quality of Data Collected PAARS Versus EODQ 

Based on the 100 point rating scale used in the study, the radio method 
high-value responses to the questions averaged 40.8, and the questionnaire 
high-value responses averaged 36.5. The diiference in favor of PAARS was not 
statistically significant. These results indicate that only moderate quality 



*Chi Square value = P< 0.05 
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information v;as collected by both methods • This suggests a real need for 
research in dev-^loping questions which are better suited for probing for infor- 
mation useful to human factors decision making, 

4. SPECIFICATIONS FOR AN OPTIMUM VOICE-RADIO SYSTEM 

The equipment procured off-the-shelf was generally satisfactory for the 
purposes of tins research. A complete description of all equipment used is con- 
tained in Appendix B. However, based upon the observations made during this 
study, the following equipment changes would result in a better system for human 
factors data-collection purposes. These changes should not degrade the present 
equipment capability as to range, battery life, selective call, etc. 

• Reduce weight and size of the personal radio unit. The 
design goal for the total package worn by the individual 
should be 25 ounces, and 25 cubic inches. 

• Provide an antenna for portable units which does not 
require adjusting by the worker nor interfere with work 
activities. A concealed or self-contained antenna system 
is desired. 

• Provide portable units with a hands-off capability for 
talk-listen after the unit is switched on. 

• Provide the base station with a capability to make 
hard-line connection between transmitter- receiver and 
a tape recorder to permit direct recording of incoming 
and outgoing communications. 

5. ADDITIONAL INFORMAL RESULTS 

As part of the exploratory test, an experimental form was developed on 
V. hich maintenance-type information was recorded. This form was filled out by 
the analyst at the base station, concurrent with his questioning of the individual 
technicians, regarding the thirteen questions listed in Table I. 
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The information contained on this experimental form was not quantified and 
summarized because the basic experiment was not designed to provide a mean- 
ingful base against which such data could be compared. It is of interest, how- 
ever, to note that maintenance time was collected using the PAARS technique 
much more rapidly than with the direct time study method. Since tabulation of 
time data on the experimental form indicates that this type information can be 
collected using much less observer time (one human factors man at the base 
station can collect as much time information as six to ten time study men), it 
appears that mere should be a very definite economic advantage in collecting 
all types of human factor information through the PAARS technique as compared 
to conventional practices. 
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SECTION IV 



CONCLUSIONS 

It is concluded that two-way voice-radio is a feasible means of collecting 
hunran performance data in operational environments. While no overall superi- 
ority was demonstrated for voice-radio versus paper-and-pencil questionnaires, 
the results suggest that a differential effect exists, with radio better for collect- 
ing information regarding equipment and tool problems. The radio data-collection 
procedure of maintenance technician initiated call^'in at significant times during 
the work day is a fruitful method of obtaining information. The radio method was. 
well received by the technician subjects. Voice-radio appears to offer another 
means of collecting human factors data in field situations, and its potential 
should be more fully explored. Future studies should be directed toward deter- 
mining the specific data-collection uses of voice-radio and attempting to measure 
the cost-effectiveness, reliability, and validity of this method as compared with 
the more conventional techniques of interviewing, observing, and questionnaire. 
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APPENDIX A 
RULES FOR JUDGES 

The judges were briefed concernir^ the general purposes of the experiment, 
and they were alerted to the aims of Personnel Subsystem Test and Evaluation 
(PSTE). As a minimum, this included the recognition that PSTE is necessary to 
evaluate whether or not personnel subsystem activities are producing the human 
performance necessary to complete systems requirements, specifically, to 
reveal and correct deficiencies in personnel selection, training, support, tech- 
nical publications, work procedures, and maintenance design problems. 

Additionally, judges were informed that the PSTE data, such as they were 
to judge, would form the basis for making decisions and formulating reports 
pertaining to the PSTE factors in a man-machine system. 

The judges were then presented in randomized order, 13 sets of 36 response 
items. The specific instructions to the Judges were: 

*'You are to choose one response of each set as providing 
maximally useful source information for PSTE, Mark this 
response on the extreme end of the scale at the position of 
**most useful." Then choose one response as minimally 
useful and place a mark at the extreme end of the scale at 
the position of * 'least useful/' Then observe the remaining 
34 response sheets of each set and determine the scale 
value to each of these responses relative to the responses 
judged as most and least useful. When you have completed 
the first set, arrange the second set, and repeat the above 
procedure, etc. Please complete each set at one sitting, 
however, you may interrupt your judgments between sets 
as you determine," 
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APPENDIX B 
EQUIPMENT USED IN STUDY 

Two kinds of equipment were used in this study: the radio communications 
equipment and the equipment for recording interviews. The communications 
equipment consisted of a receiver-transmitter with encoder unit, designated as 
the base station, and 10 portable receiver-transmitters plus a battery charger 
capable of charging all of the portable receiver-transmitters. All of the commu- 
ication units were standard commercial items manufactured by Motorola, Inc.* 

The base station receiver-transmitter was a Model P43DEN with an FM 
power output of 10 watts and operated, at 150.345 megahertz. The encoder unit 
allowed selective paging of any one of the 10 receivers. The page signal consisted 
of a loud tone on the speaker of the selected portable unit. As long as other 
portables were in the page mode of operation, they were not disturbed by the 
page signal or the transmissions between base station and the selected portable 
over the common radio frequency channel. The portable receiver-transmitters 
(commonly known as "walkie-talkies") were models H23 DEN with an FM radiated 
power of 2 watts and operated on the same frequency as the base station during 
both studies. 

Operating features, physical dimensions, and weight of the portable units 
are of some interest in this study. The weight of each portable unit with battery 
was 40 oz, and the dimensions were 1.812" x 3.375" x 9.18?" (56.06 cubic 
inches). Receiver sensitivity is specified as 0.5 microvolt for 20 db quieting. 
The portable units were worn on a "Sam Brown" Belt (Model NLN 6352A) and 
used as external speaker-microphone (Model NMN 6021B) which could be clipped 
onto the chest strap of the Sam Brown Belt. To use the portable unit to call the 
base station or to call in response to a page signal, the person wearing the unit 
had td extend the telescoping style antenna, place the page/talk switch in the 
talk position, and depress the microphone talk button. The microphone could be 
left clipped to the chest or removed and hand-held as desired. 



* Other manufacturers' equipments could have been used. 
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To record the interviews during these studies, a microphone was located to 
pickup both the voice of the base station operator and the voice reception from 
tlie base station receiver. This microphone (furnished with the tape recorder) 
was connected to a Wollensak tape recorder (Model T-1500) which was equipped 
with a footswitch to start and stop the tape. 
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